A Cost Model for Data Stream Processing on Modern Hardware

For stream processing application domains, using queries to process or analyze data incoming from potentially endless streams, low latency and high throughput are key requirements. It is not easy to achieve this as many factors influence the actual runtime of query execution plans and one can not measure all of them individually. Therefore, query optimizers try to overcome this hurdle by using cost models for decision making. Modern hardware architectures and devices, like manycore CPUs or the NVRAM storage technology demonstrate new properties for query execution, which have not received much attention within the model. Thus, traditional optimizers are not capable of dealing with these new factors leading to results possibly far away from optimum. Our work addresses this problem providing a new cost model based on modern hardware characteristics. We analyze hardware aspects necessary for query optimization and substantiate them with our own low-latency stream processing engine PipeFabric. This yields in a cost model that can precisely predict the performance of query execution plans on modern hardware close to actual measurements.